Image processing apparatus, image processing method, information processing program, fluorescence observation system, and fluorescence navigation surgery system

ABSTRACT

An information processing apparatus includes a storage unit, a cutting-out unit, and a display controller. The storage unit is configured to store an image of fluorescence emitted from an observation target area of a living body in a recording medium as a fluorescence image of the observation target area, the image of fluorescence being captured with a first definition, the image of fluorescence being obtained by applying excitation light to the observation target area to which a fluorescent reagent is added in advance. 
     The cutting-out unit is configured to cause a user to select at least a partial area of the fluorescence image and to cut out the selected area as a fluorescence image of a ROI area. The display controller is configured to cause a display unit to display the fluorescence image of a ROI area with a second definition that is lower than the first definition.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/222,162, filed Mar. 21, 2014, now U.S. Pat. No. 10,143,378, whichclaims the benefit of Japanese Priority Patent Application JP2013-074785 filed Mar. 29, 2013, the entire contents of which areincorporated herein by reference.

BACKGROUND

The present disclosure relates to a fluorescence navigation surgerysystem for performing fluorescence navigation surgery as surgery, animage processing apparatus used therefor, an image processing method, aninformation processing program, and a fluorescence observation system.

In the past, a dissection of a sentinel lymph node that is suspected tobe a metastasis has been performed in ablative surgery of a breastcancer. When the dissection of a sentinel lymph node that is not easilyidentified by visual inspection is performed, the sentinel lymph node isidentified by using indocyanine green (ICG) being a fluorescent reagent.

Examples of a method of identifying a sentinel lymph node using ICGinclude a method of applying excitation light in a near infraredwavelength band to an observation area and capturing an image offluorescence emitted from the ICG in a near infrared wavelength band toobserve the image (see, for example, Japanese Patent ApplicationLaid-open No. 2001-299676).

Examples of a commercialized system for observing an image offluorescence emitted from an ICG reagent for identifying a sentinellymph node include pde-neo (manufactured by Hamamatsu Photonics K.K.,see Japanese Patent No. 4745059), HyperEye Medical System (manufacturedby MIZUHO Co., Ltd.), and mini-FLARE (Beth Israel hospital, see Annalsof Surgical Oncology 2009 October; 16(10): 2943-2952, the Internet<URL:http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2772055/>).

The HyperEye Medical System and the mini-FLARE have functions such assuperimposing a fluorescence observation image on a visible lightobservation image captured by a built-in video camera for visible lightimage observation to display the superimposed image. In order to performthe superimposition of the images appropriately, for example, in themini-FLARE system, the video camera for visible light image observationand a video camera for fluorescence image observation are configured toshare the same zoom lens used for imaging so that the field of view ofboth images is the same, and to include a respective imaging devicehaving the same number of pixels.

Moreover, with progress in cancer gene research and reagent research inrecent years, it is becoming possible to selectively stain cancer cells(see, for example, research by Hisataka Kobayashi, Nature Medicine 15,104-109 (2009)). For example, a fluorescent reagent that causes cancercells to glow only by spraying has been developed (see the press release“fluorescent reagent that causes cancer cells to glow only by sprayinghas been developed,” [online], Nov. 24, 2011, Japan Science andTechnology Agency [searched on Mar. 18, 2013], the Internet<URL:http://www.jst.go.jp/pr/announce/20111124-2/index.html>).

Moreover, in the past, measures against a medical malpractice suit by,during surgery, capturing an image of the surgery with a video cameraand storing the image, in medical practice, have been considered. Then,an image of a surgical field with a high definition (HD) is obtained tocapture an image of the surgical technique, and images of a surgicaltable and the periphery of the surgical table (with a standarddefinition (SD) or HD) are obtained to capture an image of the whole ofthe surgery (see, for example, Japanese Patent Application Laid-open No.2005-346552).

SUMMARY

In the case where the reagent that selectively stains cancer cells isused to detect cancer cells, it is important to observe the entiresurgical field including not only an area that is focused on by anoperator during a surgical operation, i.e., an area to be removed with ascalpel or the like, but also areas other than the area. This is becausethere is a need to confirm, during or after the surgery, whether or notcancer cells that have not been removed exist.

However, the systems such as the pde-neo, the HyperEye Medical System,and the mini-FLARE are based on the assumption that the zoom lens of thevideo camera that captures an image is used to adjust the size of thefield of view observed by the operator. Specifically, a fluorescenceobservation image captured during surgery includes only an area of thesurgical field in which the operator is interested. Therefore, it hasbeen difficult to use the fluorescence observation image captured withthe system to prove that there is no unremoved cancer cell.

In addition, in the existing system, the fluorescence image is notnecessarily captured, displayed, and stored appropriately.

In view of the circumstances as described above, it is desirable toprovide an image processing apparatus, an image processing method, aninformation processing program, a fluorescence observation system, and afluorescence navigation surgery system, which are capable of capturing,displaying, and storing a fluorescence image appropriately.

According to an embodiment of the present disclosure, there is providedan information processing apparatus including a storage unit configuredto store an image of fluorescence emitted from an observation targetarea of a living body in a recording medium as a fluorescence image ofthe observation target area, the image of fluorescence being capturedwith a first definition, the image of fluorescence being obtained byapplying excitation light to the observation target area to which afluorescent reagent is added in advance, a cutting-out unit configuredto cause a user to select at least a partial area of the fluorescenceimage and to cut out the selected area as a fluorescence image of a ROIarea, and a display controller configured to cause a display unit todisplay the fluorescence image of a ROI area with a second definitionthat is lower than the first definition.

In the present disclosure, a fluorescence image of the entireobservation target area is stored with the first definition and thedisplay unit is caused to display the fluorescence image of the ROI areaselected by the user is displayed with the second definition for asurgical operation. Therefore, the two purposes of the storing and theobservation are achieved, and a fluorescence image can be captured,displayed, and stored appropriately.

In the information processing apparatus according to this embodiment,the storage unit may be configured to store RAW image data of thefluorescence image of the observation target area as it is.

According to an embodiment of the present disclosure, there is providedan image processing method including storing an image of fluorescenceemitted from an observation target area of a living body in a recordingmedium as a fluorescence image of the observation target area, the imageof fluorescence being captured with a first definition, the image offluorescence being obtained by applying excitation light to theobservation target area to which a fluorescent reagent is added inadvance, causing a user to select at least a partial area of thefluorescence image and cutting out the selected area as a fluorescenceimage of a ROI area, and causing a display unit to display thefluorescence image of a ROI area with a second definition that is lowerthan the first definition.

According to an embodiment of the present disclosure, there is providedan information processing program that causes a computer to function asa storage unit configured to store an image of fluorescence emitted froman observation target area of a living body in a recording medium as afluorescence image of the observation target area, the image offluorescence being captured with a first definition, the image offluorescence being obtained by applying excitation light to theobservation target area to which a fluorescent reagent is added inadvance, a cutting-out unit configured to cause a user to select atleast a partial area of the fluorescence image and to cut out theselected area as a fluorescence image of a ROI area, and a displaycontroller configured to cause a display unit to display thefluorescence image of a ROI area with a second definition that is lowerthan the first definition.

According to an embodiment of the present disclosure, there is provideda fluorescence observation system including an excitation light sourceconfigured to apply excitation light to an observation target area of aliving body to which a fluorescent reagent is added in advance, animaging unit configured to capture an image of fluorescence emitted fromthe observation target area with a first definition as a fluorescenceimage of the observation target area, a storage unit configured to storethe fluorescence image of the observation target area in a recordingmedium, a cutting-out unit configured to cause a user to select at leasta partial area of the fluorescence image and to cut out the selectedarea as a fluorescence image of a ROI area, and a display unitconfigured to display the fluorescence image of a ROI area with a seconddefinition that is lower than the first definition.

According to an embodiment of the present disclosure, there is provideda fluorescence navigation surgery system including an excitation lightsource configured to apply excitation light to an observation targetarea of a living body to which a fluorescent reagent is added inadvance, a fluorescence imaging unit configured to capture an image offluorescence emitted from the observation target area with a firstdefinition as a fluorescence image of the observation target area, astorage unit configured to store the fluorescence image of theobservation target area in a recording medium, a cutting-out unitconfigured to cause a user to select at least a partial area of thefluorescence image and to cut out the selected area as a fluorescenceimage of a ROI area, a fluorescence image display unit configured todisplay the fluorescence image of a ROI area with a second definitionthat is lower than the first definition, a visible light imaging unitconfigured to cause the user to select at least a partial area of theobservation target area and to capture an image of the selected area asa visible light image, and a visible light image display unit configuredto display the visible light image.

According to an embodiment of the present disclosure, there is provideda fluorescence navigation surgery system including an excitation lightsource configured to apply excitation light to an observation targetarea of a living body to which a fluorescent reagent is added inadvance, a fluorescence imaging unit configured to capture an image offluorescence emitted from the observation target area with a firstdefinition as a fluorescence image of the observation target area, astorage unit configured to store the fluorescence image of theobservation target area in a recording medium, a visible light imagingunit configured to select at least a partial area of the observationtarget area as a ROI area based on information on a selected positioninput by a user and to capture a visible light image of the ROI area, acutting-out unit configured to cut out a fluorescence image of the ROIarea from the fluorescence image of the observation target area based onthe information on a selected position, a superimposing unit configuredto generate a superimposed image by aligning a position of the visiblelight image with a position of the cut fluorescence image andsuperimposing the images, and a superimposed image display unitconfigured to display the superimposed image with a second definitionthat is lower than the first definition.

According to an embodiment of the present disclosure, there is provideda fluorescence navigation surgery system including an excitation lightsource configured to apply excitation light to an observation targetarea of a living body to which a fluorescent reagent is added inadvance, a fluorescence imaging unit configured to capture an image offluorescence emitted from the observation target area with a firstdefinition as a fluorescence image of the observation target area, astorage unit configured to store the fluorescence image of theobservation target area in a recording medium, a visible light imagingunit configured to cause a user to select at least a partial area of theobservation target area and to capture a visible light image of theselected area as a ROI area, a cutting-out unit configured to cut out afluorescence image of the ROI area from the fluorescence image of theobservation target area based on edge information included in thevisible light image, a superimposing unit configured to generate asuperimposed image by aligning a position of the visible light imagewith a position of the cut fluorescence image and superimposing theimages, and a superimposed image display unit configured to display thesuperimposed image with a second definition that is lower than the firstdefinition.

As described above, according to the present disclosure, it is possibleto capture, display, and store a fluorescence image appropriately.

These and other objects, features and advantages of the presentdisclosure will become more apparent in light of the following detaileddescription of best mode embodiments thereof, as illustrated in theaccompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a whole configuration diagram of a fluorescence navigationsurgery system 100 according to a first embodiment;

FIG. 2 is a diagram showing an image of an entire surgical field SA inwhich only a fluorescence area FA glows;

FIG. 3 is a diagram showing functional blocks of an image processingapparatus 10B according to a modified example;

FIG. 4 is a whole configuration diagram of a fluorescence navigationsurgery system 100B according to a second embodiment; and

FIG. 5 is a whole configuration diagram of a fluorescence navigationsurgery system 100C according to a third embodiment.

DETAILED DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present disclosure will be describedwith reference to the drawings.

First Embodiment

(Entire Configuration)

The entire configuration of a fluorescence navigation surgery systemaccording to a first embodiment will be described first. FIG. 1 is awhole configuration diagram of the fluorescence navigation surgerysystem 100 according to this embodiment.

The fluorescence navigation surgery system 100 is largely divided into avisible light capturing system 300 for an operator to observe thesurgical field SA and a fluorescence capturing system 200 that captures,stores, and displays a fluorescence image. Since the fluorescencenavigation surgery system 100 is divided into the visible lightcapturing system 300 and the fluorescence capturing system 200, zoomlenses of the capturing systems can be operated independently unlike theabove-mentioned system such as the mini-FLARE, which shares the samezoom lens.

(Visible Light Capturing System)

The visible light capturing system 300 is configured to include a videocamera for surgical field observation 1, an optical zoom lens 2, and amonitor for surgical field observation 3. The video camera for surgicalfield observation 1 is capable of capturing an image of the entiresurgical field SA, and the capturing direction thereof can be rotated,i.e., the video camera can be panned. Further, the video camera forsurgical field observation 1 can perform optical zooming with theoptical zoom lens 2 to capture an image of only an area ROI in which theoperator is interested. Because the video camera for surgical fieldobservation 1 needs to capture an image of the operator's handling of aneedle accurately, it is expected to output an image at a high framerate. It should be noted that the area ROI in which the operator isinterested represents an area he/her is thinking about trying to removeor an area that has been removed.

The monitor for surgical field observation 3 is configured to display animage with an HD. As shown in FIG. 1, the entire surgical field SA isdisplayed on the monitor for surgical field observation 3, for example.In the case where the fluorescence wavelength is a near-infraredwavelength that is not in a visible light band, the fluorescence area FAis not displayed on the monitor for surgical field observation 3. Itshould be noted that the visible light capturing system 300 is usedmainly in the case of endoscopic surgery and is not used in the case ofgeneral abdominal surgery or open heart surgery in some cases.

(Fluorescence Capturing System)

On the other hand the fluorescence capturing system 200 is configured toinclude an LED 4, a lens 5, an exciting filter 6, a 24M pixel camera 7,an optical zoom lens 8, an emission filter 9, an image processingapparatus 10, and a monitor for fluorescence image observation 16. TheLED 4, the lens 5, and the exciting filter 6 constitute an excitationlight source. It should be noted that the fluorescence capturing system200 may have the same size and shape as that of the above-mentionedpde-neo or mini-FLARE in the case of general abdominal surgery or openheart surgery, and may have a size that can be integrated into alaparoscope or an endoscope in the case of laparoscopic surgery orendoscopic surgery, respectively.

Moreover, the image processing apparatus 10 is configured to include adefect correction and development unit 11, an image compression unit 12,an image storage unit 13, the digital zooming and panning unit 14, and agraphic board 15.

The image processing apparatus 10 is constituted of hardware elements ofa personal computer (PC) including, for example, a CPU (CentralProcessing Unit), a ROM (Read Only Memory), and a RAM (Random AccessMemory). Alternatively, the image processing apparatus 10 may beconstituted of a dedicated IC such as an FPGA (Field Programmable GateArray).

The image processing apparatus 10 exchanges various signals with thedefect correction and development unit 11, the image compression unit12, the image storage unit 13, the digital zooming and panning unit 14,and the graphic board 15 and executes various types of arithmeticprocessing to process a fluorescence image. Various programs and varioustypes of data for the various types of arithmetic processing are loadedto the RAM. The CPU executes the programs loaded to the RAM. The ROMstores the programs and data loaded to the RAM.

It should be noted that the CPU executes the program loaded to the RAMfrom the ROM or the like, and thus functional blocks of the defectcorrection and development unit 11, the image compression unit 12, theimage storage unit 13, and the digital zooming and panning unit 14 areachieved.

(Details of Elements and Flow of Processing)

First, in the fluorescence capturing system 200, excitation light isapplied to the surgical field SA by LED illumination including the LED4, the lens 5, and the exciting filter 6.

A fluorescent contrast medium such as ICG is locally injected or sprayedin the surgical field SA in advance and thus is collected at the desiredplace. The excitation light applied to the surgical field SA causes thefluorescent contrast medium to glow (hereinafter, the area that glows,on which the fluorescent contrast medium collects, being referred to asfluorescence area FA).

The fluorescence emitted from the fluorescence area FA is transmittedthrough the emission filter 9, and an image of the fluorescence isenlarged by the optical zoom lens 8 and is captured by the 24M pixelcamera 7. The emission filter 9 plays a role of blocking light having awavelength other than that of the target fluorescence. The optical zoomlens 8 plays a role of adjusting the field of view by the 24M pixelcamera 7 so that an image of an entire image storing area RA that mayundergo surgery is captured.

The 24M pixel camera 7 is the most important element in the presentdisclosure, has 24M pixels, and includes an imaging device that iscapable of capturing an image with an ultra-high definition of 6 k×4 k,for example. From the imaging device, an image can be output at the rateof up to 6.3 fps, for example. Because the 24M pixel camera 7 does notcapture an image of a movable object such as handling of a needle butcapture an image of an object that does not move such as cancer cellsemitting fluorescence unlike the video camera for surgical fieldobservation 1, it only needs to have a frame rate from 1 fps to 3 fps.

The fluorescence image output from the 24M pixel camera 7 is input tothe image processing apparatus 10. The fluorescence image input to theimage processing apparatus 10 is subject to development processing suchas defect correction processing and demosaic processing in the defectcorrection and development unit 11, first. After the developmentprocessing, the fluorescence image is output to the image compressionunit 12 and the digital zooming and panning unit 14.

The fluorescence image input to the image compression unit 12 iscompressed through image compression processing and thus the image sizeis reduced. After that, the compressed fluorescence image is output tothe image storage unit 13.

After the image compression, the fluorescence image input to the imagestorage unit 13 is stored in the image storage unit 13. The imagestorage unit 13 includes a storage medium such as an HDD (hard diskdrive), and the compressed fluorescence image is stored in the storagemedium. The image stored in the storage medium is, for example, an imageof the entire surgical field SA in which only the fluorescence area FAglows, which is shown in FIG. 2.

After the development processing, the digital zooming and panning unit14 performs digital zooming processing or panning processing on thefluorescence image output to the digital zooming and panning unit 14 toenlarge an image of only the area ROI in which the operator isinterested and to display the image.

Further, the digital zooming and panning unit 14 may perform smallpixilation processing to clearly display a fluorescence image of onlythe area ROI in which the operator is interested, i.e., display theimage pixel by pixel on the monitor for fluorescence image observation16.

The fluorescence image that has been subject to the processing in thedigital zooming and panning unit 14 is displayed on the monitor forfluorescence image observation 16 via the graphic board 15. The monitorfor fluorescence image observation 16 is configured to display an imagewith an HD. As shown in FIG. 2, on the monitor for fluorescence imageobservation 16, the area ROI including the fluorescence area FA, inwhich the operator is interested, is enlarged and displayed, forexample.

(Fluorescence Image to be Stored and Fluorescence Image to be Displayed)

The present disclosure has a feature that an image stored to prepare fora medical malpractice suit or the like is different from an imagedisplayed on the monitor for fluorescence image observation 16 to bereferred to by an operator during a surgical operation. Specifically,the area ROI in which the operator is interested is displayed on themonitor for fluorescence image observation 16, but the image stored inthe image storage unit 13 is obtained by capturing an image of theentire surgical fields SA, in which the field of view is adjusted by theoptical zoom lens 8.

The image stored in the image storage unit 13 is obtained not bycapturing an image of only the area ROI in which the operator isinterested in at the time of surgery but by capturing a fluorescenceimage of the entire surgical field SA. Therefore, after surgery, it ispossible to confirm whether or not there is an unremoved portion of thearea including, for example, cancer, which is fluorescent-labeled duringthe surgery.

(Comparison with Typical Technique)

In the above-mentioned typical technique, the imaging device included inthe camera that captures an image has a high definition of 2M pixelseven if the imaging device is a high-definition one. For that reason,even if an image of the entire surgical field SA is enlarged by digitalzooming and only the area ROI in which the operator is interest isextracted as in the present disclosure, the image quality isdeteriorated and thus the image is not suitable for practical use.Therefore, it has no choice but to capture an image of only the area inwhich the operator is interested during surgery, and it may beimpossible to capture an image of the entire surgical field SA forconfirming whether there is an unremoved portion including, for example,cancer after surgery.

Moreover, in the case where an image of the surgical field SA having asize of, for example, about 15 cm×10 cm is captured using an imagingdevice having about 2M pixels, the pixel resolution of the imagingdevice is about 75 μm. Specifically, because the resolution is severaltimes larger than the size of a cell, cancer cells are highly likely tobe missed in the confirmation of unremoved cancer cells.

However, in the case where the 24M pixel imaging device used in theembodiment of the present disclosure is used to capture an image of thesurgical field SA having a size of about 15 cm×10 cm, the pixelresolution of the imaging device is about 25 μm. Because the resolutionof almost the same size of a cancer cell can be achieved, it is possibleto store a fluorescence image with a definition in which cancer cells orthe like fluorescent-labeled are hard to be missed.

Moreover, in this embodiment, an image of the surgical field SA having asize of, for example, about 15 cm×10 cm captured using the 24M pixelimaging device is subject to digital zooming processing, and the areaROI in which the operator is interested having a size of, for example,about 5 cm×2.5 cm is enlarged and displayed. Even in this case, becausethe image can be displayed on the monitor for fluorescence imageobservation 16 with a definition of not less than a pixel by pixeldefinition, it is possible to provide a fluorescence image withfavorable quality to the operator.

(Regarding Size of Surgical Filed and Number of Pixels of ImagingDevice)

Now, the relationship between the size of the surgical field SA whoseimage is captured and the number of pixels of the imaging device thatcaptures a fluorescence image will be described.

For example, in an ovariectomy using a laparoscope in obstetrics andgynecology department, because the observation field of view is adjustedso that a clamp and a needle used in the handling of a needle do notmove out of the field of view, the field of view whose image is capturedis generally set to be about 7 to 8 cm square or a little more thanthat.

Then, in order not to miss a tumor having a size of about 100 μm, it isdesirable to use an optical system having a pixel resolution of about 50μm or less to capture a fluorescence image.

In the case where an image of the surgical filed SA having a size of 7cm on a side with a pixel resolution of 50 μm is captured, the number ofpixels of the imaging device is 1400. Therefore, in the case of surgicalfield SA having a size of about 10 cm on a side, it is desirable to usean imaging device having not less than 2000 pixels on the short side inorder to achieve the above-mentioned effects of the present disclosure.

(Regarding Effects of Present Disclosure)

The fluorescence navigation surgery system according to this embodimentis capable of enlarging an image of the area ROI in which the operatoris interested by digital zooming in response to a user's request anddisplaying the image, and capturing and storing an image of the entiresurgical filed SA with a pixel resolution in which a tumor is notmissed. Therefore, it is possible to capture an image of an area inwhich the operator is not interested, which is not captured in thetypical system, and to store the image. Therefore, it is possible toprepare for a market of countermeasures against a medical malpracticesuit, for example.

(Modified Example (Regarding Configuration in which Image is notCompressed when being Stored))

Next, a modified example of the method of storing, in the image storageunit 13, a fluorescence image captured by the 24M pixel camera 7 will bedescribed.

In the above-mentioned embodiment, the fluorescence image is developedin the defect correction and development unit 11 and then is compressedin the image compression unit 12. However, in the case where thedeterioration of the image due to the compression is expected to beavoided, the fluorescence image is not compressed and is stored. In thisregard, in this modified example, RAW image data output from the 24Mpixel camera 7 is not compressed and is stored in the image storage unit13.

In general, in a color imaging device in which color filters of colorsof red, green, and blue (RGB) are arranged in a Bayer pattern on asingle plate, RAW image data is output, the output RAW image data isdeveloped, and thus a final color image can be obtained. The size of thecolor image obtained through the development processing is significantlylarger than that of the original RAW image data.

For example, the size of RAW image data output from an imaging devicehaving 24M pixels is about 50 MB in the case where luminance informationis 14 bits. If the RAW image data is developed into a color image whoseluminance information is 16 bits, the size is 147 MB. Then, even if theluminance information of the developed color image is reduced to 8 bits,the image still has a size of about 73 MB.

As described above, the original RAW image data has a smaller size thanthe color image whose luminance information is reduced from 16 bits to 8bits. Therefore, in the case where the image is not compressed and isstored, it is possible to store the image with a small data size whenthe RAW image data before development is stored as it is. Then, becausethe capacity of the storage medium that is necessary to store afluorescence image can be reduced in the image storage unit 13, it isefficient from a view point of a cost of system operation.

FIG. 3 shows functional blocks of the image processing apparatus 10Baccording to this modified example. As shown in FIG. 3, the imageprocessing apparatus 10B is configured to include a defect correctionunit 11B, the image storage unit 13, a development unit 11C, the digitalzooming and panning unit 14, and the graphic board 15.

The defect correction unit 11B and the development unit 11C are obtainedby dividing functions of the defect correction and development unit 11according to the above-mentioned embodiment into defect correctionprocessing and development processing. The image processing apparatus10B does not include the image compression unit 12.

Regarding a flow of processing, the image processing apparatus 10Breceives a fluorescence image output from the 24M pixel camera 7 first.

The fluorescence image input to the image processing apparatus 10B issubject to defect correction processing in the defect correction unit11B. After the defect correction processing, the fluorescence image isoutput to the image storage unit 13 and the development unit 11C.

The fluorescence image input to the image storage unit 13 is stored inthe image storage unit 13. The function and configuration of the imagestorage unit 13 is the same as that in the above-mentioned embodiment.

The fluorescence image input to the development unit 11C is subject todevelopment processing such as demosaic processing. After that, thedeveloped fluorescence image is output to the digital zooming andpanning unit 14.

After the development processing, the digital zooming and panning unit14 performs digital zooming processing or panning processing on thefluorescence image output to the digital zooming and panning unit 14 inorder to enlarge an image of only the area ROI in which the operator isinterested and display the image. The function and configuration of thedigital zooming and panning unit 14 is the same as that in theabove-mentioned embodiment.

The fluorescence image that has been subject to processing in thedigital zooming and panning unit 14 is displayed on the monitor forfluorescence image observation 16 via the graphic board 15.

Second Embodiment

Next, a second embodiment of the present disclosure will be described.The major difference between the second embodiment and the firstembodiment is that the monitor for surgical field observation 3 isseparated from the monitor for fluorescence image observation 16 in thefirst embodiment and two images are superimposed and the superimposedimage is observed on a monitor in the second embodiment. It should benoted that the second embodiment is the same as the first embodiment inthat the area ROI in which the operator is interested is displayed on amonitor and an image of the entire image storing area RA that mayundergo surgery is stored.

FIG. 4 shows the entire configuration of the fluorescence navigationsurgery system 100B according to this embodiment. It should be notedthat elements of the same functions as those according to the firstembodiment will be denoted by the same reference symbols and adescription thereof will be omitted.

The fluorescence navigation surgery system 100B is largely divided intoa visible light capturing system 300B for the operator to observe thesurgical field SA and a fluorescence capturing system 200B that capturesand stores a fluorescence image. Then, a system for surgery based on theassumption that surgery is carried out while seeing an image forsurgical field observation captured by the visible light capturingsystem 300B is assumed. Therefore, the fluorescence navigation surgerysystem 100B further includes an image processing apparatus 10D thataligns a display range of a fluorescence image captured by thefluorescence capturing system 200B with a display range of an¥¥the imagefor surgical field observation to superimpose the images and a monitorfor superimposed image observation 16B that displays the superimposedimage.

(Visible Light Capturing System)

The visible light capturing system 300B is configured to include thevideo camera for surgical field observation 1 and the optical zoom lens2. The video camera for surgical field observation 1 is capable ofcapturing an image of the entire surgical field SA, and the capturingdirection thereof can be rotated, i.e., the video camera can be panned.Further, the video camera for surgical field observation 1 is capable ofperforming optical zooming with the optical zoom lens 2 to capture animage of only the area ROI in which the operator is interested. Theimage captured by the video camera for surgical field observation 1 isoutput to the image processing apparatus 10D.

(Fluorescence Capturing System)

On the other hand, the fluorescence capturing system 200B is configuredto include the LED 4, the lens 5, the exciting filter 6, the 24M pixelcamera 7, the optical zoom lens 8, the emission filter 9, and an imageprocessing apparatus 10C.

Moreover, the image processing apparatus 10C is configured to includethe defect correction and development unit 11, the image compressionunit 12, and the image storage unit 13 as functional blocks.

The fluorescence image that has been subject to defect correctionprocessing and development processing in the defect correction anddevelopment unit 11 is output to the image compression unit 12 and thedigital zooming and panning unit 14 of the image processing apparatus10D.

(Configuration of Image Processing Apparatus 10D and Flow of Processing)

The image processing apparatus 10D is configured to include functionalblocks of the digital zooming and panning unit 14 and a positionalignment and image combining unit 17, and the graphic board 15. Itshould be noted that the image processing apparatuses 10C and 10D may beconstituted of a PC or a dedicated IC such as an FPGA similarly to theimage processing apparatus 10 according to the first embodiment.

In the image processing apparatus 10D, the digital zooming and panningunit 14 receives a fluorescence image supplied from the defectcorrection and development unit 11 and information on optical zoomingand panning (position selecting information) of the video camera forsurgical field observation 1 supplied from the video camera for surgicalfield observation 1, first. The digital zooming and panning unit 14performs digital zooming processing and digital panning processing onthe fluorescence image based on the received information on opticalzooming and panning to appropriately superimpose the fluorescence imagewith the image supplied from the video camera for surgical fieldobservation 1. The fluorescence image that has been subject to zoomingprocessing and panning processing is output to the position alignmentand image combining unit 17.

Next, the position alignment and image combining unit 17 superimposesthe image for surgical field observation supplied from the video camerafor surgical field observation 1 with the fluorescence image suppliedfrom the digital zooming and panning unit 14 and combines the images.The superimposed image thus combined is output to the monitor forsuperimposed image observation 16B via the graphic board 15 and isdisplayed on a screen with an HD.

It should be noted that in the above description, position alignment isperformed based on the information on optical zooming and panning of thevideo camera for surgical field observation 1, which is supplied fromthe video camera for surgical field observation 1. However, the presentdisclosure is not limited thereto, and the position for superimposingimages is obtained by image analysing processing such as edge detectionto superimpose the images.

Hereinabove, the configuration of the second embodiment has beendescribed. It should be noted that in the above description, theconfiguration in which an image for surgical field observation and afluorescence image are superimposed and the image thus obtained isdisplayed on the monitor for superimposed image observation 16B has beendescribed. However, there is a request that displays the image forsurgical field observation and the fluorescence image separately on amonitor during surgery depending on the operator's taste. Therefore,also in the configuration of the second embodiment, it is possible toseparately extract the image for surgical field observation output fromthe video camera for surgical field observation 1 and output to themonitor for surgical field observation 3 and to separately extract thefluorescence image output from the digital zooming and panning unit 14and output to the monitor for fluorescence image observation 16.

Third Embodiment

Next, a third embodiment of the present disclosure will be described.The major difference between the third embodiment and the secondembodiment is that the video camera for surgical field observation 1 andthe 24M pixel camera 7 capture an image of the image storing area RA orthe area ROI in which the operator is interested from differentdirections in the second embodiment and the image is captured from thesame optical axis direction in the third embodiment.

FIG. 5 is a whole configuration diagram of the fluorescence navigationsurgery system 100C according to this embodiment. It should be notedthat elements of the same functions as those according to the firstembodiment and the second embodiment will be denoted by the samereference symbols and a description thereof will be omitted.

As described above, the fluorescence navigation surgery system 100C is acapturing system in which a visible light capturing system is integratedwith a fluorescence capturing system and a part of an optical system isshared.

(Configuration and Flow of Capturing)

The capturing system is configured to include the video camera forsurgical field observation 1, an optical zoom lens 2B, a lens drive unit21, a panning stage 20, the LED 4, the lens 5, the exciting filter 6,the 24M pixel camera 7, an optical zoom lens 8C, the emission filter 9,a dichroic mirror 9B, a zoom lens 8B, a visible light LED 30, a lens 31,and a visible light filter 32.

Capturing with visible light will be described first. Visible lightillumination including the visible light LED 30, the lens 31, and thevisible light filter 32 applies visible light to the image storing areaRA. Light that has been reflected from the image storing area RA isincident on the zoom lens 8B. The field of view of the zoom lens 8B isadjusted in the 24M pixel camera 7 in such a way that an image of theentire image storing area RA is captured. Visible light that hastransmitted through the zoom lens 8B is incident on the dichroic mirror9B. The dichroic mirror 9B is configured to reflect only fluorescenceemitted from the fluorescence area FA and cause visible light to betransmitted therethrough. The visible light that has transmitted throughthe dichroic mirror 9B is incident on the zoom lens 2B. The zoom lens 2Bis driven by the lens drive unit 21 and the field of view of the videocamera for surgical field observation 1 is enlarged at a magnificationthat is necessary for capturing an image of the area ROI in which theoperator is interested. Moreover, the field of view of the video camerafor surgical field observation 1 is focused on the area ROI in which theoperator is interested by panning the video camera for surgical fieldobservation 1 by the panning stage 20.

In this way, an image of the area ROI in which the operator isinterested is captured in the video camera for surgical fieldobservation 1. The captured visible light image is output to theposition alignment and image combining unit 17. Moreover, in order tofocus the field of view on only the area ROI in which the operator isinterested, information on zooming performed by the lens drive unit 21and panning performed by the panning stage 20 is output to the digitalzooming and panning unit 14 as information on optical zooming andpanning (position selecting information).

Next, capturing with fluorescence will be described. The fluorescencearea FA glows with excitation light emitted from an excitation lightsource including the LED 4, the lens 5, and the exciting filter 6. Thefluorescence emitted from the fluorescence area FA is transmittedthrough the zoom lens 8B whose field of view is adjusted in such a waythat an image of the entire image storing area RA is captured, isreflected by the dichroic mirror 9B, is transmitted through the emissionfilter 9 and the optical zoom lens 8C, is incident on the 24M pixelcamera 7, and is captured.

In this way, a fluorescence image of the entire image storing area RA iscaptured by the 24M pixel camera 7. The captured fluorescence image isoutput to the defect correction and development unit 11.

With the above-mentioned configuration, it is possible to capture avisible light image and a fluorescence image from the same direction.Therefore, the position alignment in the position alignment and imagecombining unit 17 can be easily performed, and it is possible toaccurately align positions of two types of images and to superimpose theimages.

Hereinabove, the third embodiment has been described.

(Supplementary Note)

In addition, embodiments of the present technology are not limited tothe above-mentioned embodiments and various modifications can be madewithout departing from the gist of the present technology.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

What is claimed is:
 1. A surgical navigation system for use duringsurgical operation, comprising: a surgical microscope including: atleast one light source device configured to apply excitation light to anobservation target of a living body; a fluorescent imaging deviceconfigured to capture a fluorescent image of the observation targetbased on excitation light reflected from the observation target, thefluorescent image having a first pixel resolution; and a visible lightimaging device configured to capture a visible light image of theobservation target, the visible light imaging device having a red,green, and blue (RGB) color filter pattern and generating the visiblelight image as a color image according to a demosaicing process; adisplay having a second pixel resolution different from the first pixelresolution; and circuitry configured to: receive selecting informationabout a partial area of the observation target that corresponds to adefect corrected and uncompressed region of interest (ROI), theselecting information including information on at least one of zoomingor panning performed by the visible light imaging device when capturingthe visible light image of the observation target; select a portion ofthe fluorescent image based on the selecting information; generate asuperimposed image by aligning the selected portion of the fluorescentimage with the visible light image; and control the display to displaythe superimposed image with the ROI.
 2. The surgical navigation systemof claim 1, further comprising: an optical device that directs theexcitation light toward the fluorescent imaging device and that directsthe visible light toward the visible imaging device.
 3. The surgicalnavigation system of claim 2, wherein the optical device reflects theexcitation light and passes the visible light.
 4. The surgicalnavigation system of claim 2, wherein the optical device comprises adichroic mirror.
 5. The surgical navigation system of claim 1, whereinthe selecting information includes edge information of the visible lightimage.
 6. The surgical navigation system of claim 1, further comprising:a first zoomable lens optically coupled to the visible light imagingdevice; and a second zoomable lens optically coupled to the fluorescentimaging device.